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Abstract — We revisit the error correction scheme of real- valued 
signals when the codeword is corrupted by gross errors on a 
fraction of entries and a small noise on all the entries. Combining 
the recent developments of approximate message passing and the 
spatially-coupled measurement matrix in compressed sensing we 
show that the error correction and its robustness towards noise 
can be enhanced considerably. We discuss the performance in the 
large signal limit using previous results on state evolution, as well 
as for finite size signals through numerical simulations. Even for 
relatively small sizes, the approach proposed here outperforms 
convex-relaxation-based decoders. 

I. Introduction 

Although information is discrete in the classical coding the- 
ory, there are situations of interest where one should consider 
real- valued signals, such as scrambling of discrete time analog 
signals for privacy [1], network 0, (3] or jointed source 
and channel coding [4], or in the impulse noise cancellation 
in orthogonal frequency division multiplexing system Q. To 
perform error correction for such real signals over a channel 
with a gross errors on a fraction of elements (and small noise 
for all of them) a compressed-sensing-based scheme has been 
proposed by Donoho and Huo (6) and Candes and Tao (7). 
Here we reconsider this problem, taking full advantage of the 
recent progresses in compressed sensing theory. 

The problem is easily stated. One is given a real-valued 
signal x, and a channel that adds gross errors to a fraction 
of elements. Is there a way to encode the signal such that 
the errors added by the channel can be corrected? Can this 
approach still be used when the channel is in addition adding a 
small noise to all elements (a situation arguably much closer to 
some real channels |6], |8|, [9])? The method proposed in [6], 
0, is to first multiply the signal x by a random matrix in 
order to create a codeword of larger dimension, and then to use 
the classical compressed sensing approach, based on convex- 
relaxation decoding, to correct the errors of transmission. 

Our contribution in the present paper is three-fold. (1) 
We replace the convex-relaxation decoding by the Bayesian 
Approximate Message-Passing (AMP) decoder that uses the 
available prior information about the error vector |[T0lL ifTTTl . 
This provides a significant improvement in performances. (2) 
We consider a quasi-sparse channel where, in addition to the 
gross errors on a fraction of elements, there is a small additive 
random white noise, and in the lines of |12| show that the 



performance of AMP decoder is stable under this additional 
noise. (3) We use spatially-coupled measurement matrices in 
the decoding ifTTIl . |[T3lL which allows to further enhance the 
possibility for error correction (and up to its information- 
theoretical limit in the case of strictly sparse noise). 

Our paper relies on the development of the Bayesian AMP 
algorithm |[TQl , ifTTl . fBll . 02L whose behavior for large 
signal sizes can be studied rigorously using the state evo- 
lution technique |[T6lL IfTTIl ; on the development of spatially 
coupled error correcting codes on binary variables, see |[T8l 
for a review, and related measurement matrices in compressed 
sensing IfTTIl . |[T3lL lfT9l ; and on the use of quasi-sparsity 
in the Bayesian approach [12]. While the analysis of the 
reconstruction performance of AMP are rigorous in the large 
signal limit, we also consider the performance for finite size 
signals through numerical experiments. 

II. Compressed-sensing Based Error-Correction 

Consider a real- valued vector of information x G M. N , 
encode this vector by a full-rank real M x N matrix A, 
with 7 = M/N > 1 being the coding rate (the redundancy 
rate introduced in the code), so that the encoded vector is 
y = Ax £ M M . Since A is full rank, one can recover the 
original signal x from the encoded one multiplying by the 
pseudo-inverse. The encoded signal is sent through a noisy 
channel and gives rise to the corrupted codeword y = y + e 
where the element of e are iid with a distribution 



P(e») = a/V(0, 1 + e) + (1 - p)AT(p, e) , 



(D 



where < p < 1. We thus have a fraction p of elements with 
gross (variance 1 + e) errors, the rest having small (variance 
e) amplitudes. One then considers a full rank "parity-check"- 
like matrix F such that FA = 0. We construct such a pair of 
matrices by first choosing a RxM matrix F with independent 
normally distributed elements of zero mean and variance \jM 
(or variance specified by the seeding matrix, see later), the 
kernel of F is then the range of the encoding matrix A F] One 
must have R < M — N, in order to minimize the coding rate 

! Note that |7|, |8| take A as the random matrix, we choose the opposite 
in order to be able to implement the spatially coupled decoding. 
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Fig. 1. Phase diagram showing the encoding rate 7 = M/N necessary 
to perform error correction over a channel with noise described by eq. ^, 
plotted as a function of the noise sparsity p, zoom in the inset. The black (top) 
curve 7dt depicts the limit of performance of the t\ -minimization approach 
for e = 0. The blue (2nd from top) curve shows limit of performances of the 
Bayesian AMP approach for e = 0, note that up to about e < 10 ~ 5 this curve 
does not change visibly. The green (bottom) curve, given by 7 op t = 1/(1— p), 
depicts the lowest possible encoding rate for which exact decoding is possible 
for e = 0. Above the red (3rd from top) line error correction with MSE 
comparable to e = 10 -6 is possible with the Bayes optimal estimation of the 
error vector. These two rates 7 op t can be reached in the limit of large signal 
size using the seeded Bayesian AMP approach. 



j = M/N, we take from now on R = M — N. The application 
of F to the corrupted signal y results in the real- valued vector 



h = F(y + e) = Fe , 



(2) 



where h has dimension M — N and e is a quasi- sparse vector 
of dimension M. This is a compressed sensing problem: 
reconstruct the quasi- sparse M-dimensional error e given 
M — TV of its linear projections (measurements) h. In the 
context of compressed sensing F is the measurement matrix. 
Let us first review the possibility of this error-correction 
scheme when the error e is exactly sparse, i.e. e = 0. Using an 
intractable £0 minimization, the gross error e in eq. ^ can be 
found exactly as long asM-iV> Mp. So error correction in 
real valued signals corrupted by sparse gross noise is possible 
(but hard) for encoding rates 7 > 7 op t = 1/(1 — /?). Popular 
tractable l\ minimization, as used in [7], |8 ], recovers the error 
e exactly when M — N > qjdt^, where c^dt is the Donoho- 
Tanner measurement rate (20). This means that the coding 
rate must be larger than 7 > 7dt = 1/(1 — <^dt)- These two 
transitions are depicted in Fig. [T] and one can see that 7dt is 
considerably larger than 7 opt . A first step to improvement is 
to decode with an approximate message passing approach. 

III. Decoding with approximate message passing 

Consider the compressed sensing problem in eq. ^ to 
reconstruct the vector e based on the knowledge of the M—N- 
dimensional h and the matrix F. When the distribution of the 
error elements are known, the Bayes optimal way of estimating 
e that minimizes the MSE E = YnLi( e i ~ &i) 2 l M with the 



true error e is given as 



/ 



AeieiViiei) , 



(3) 



where z^(e^) is the marginal probability distribution of the 
variable i 

i/t(e»)= / P(e\h)Y[d ej 



(4) 



under the posterior measure 



M-N M 

p(e|h) = z(h) p{e) n 5 ^ -j2 p ^ 



(5) 



Such an optimal Bayes reconstruction is not computationally 
tractable in general and in order to get an estimate of the 
marginals z^(e^), we use the AMP algorithm. 

A. AMP reminder 

We define, for every component /i of h: V^ and cj m , and 
for each component i of e: X^, R i9 a^ V{. These quantities are 
updated iteratively as follows (for the derivation in the present 
notation see |[T5) and for the original one [14], |[T0l ) 
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Where, using F(e) = JJ i P{ei) as given in eq. {II, the 
functions f a and f c read lfl2l 



fa(£\R) = 



sr^2 

l2a=l W a< 



2^+a 2 .) 



Rat 



(S2+a2)2 



(12) 
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(15) 



The initialization is set as: a 



t=o 



0, v] 



t=0 



-e, uj 






hjj,. Once the convergence of the iterative procedure is obtained 
the estimate of the ith component of the error e is a\. 



B. Performance of AMP 

As shown by Bayati and Montanari (T6), in the limit of large 
system sizes, i.e. when parameters p, e, 7 are fixed whereas 
M — )> 00, the evolution of the AMP algorithm can be described 
exactly using the "state evolution". This allows to evaluate 
the MSE achieved by the AMP reconstruction. When F is an 
homogeneous random iid matrix the state evolution is written 
in terms of MSE at iteration-time t, which evolves as (for a 
derivation see e.g. lfT4l . |[T5lL lfT6l ) 




e 
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■ : unit coupling 

: coupling/ 
w : interaction range 
□ : no coupling (null elements) 
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m 



a 

(16) 

where Vz = dze~ z ^ 2 /V2tt is a Gaussian measure for the 
integral, and where E t=0 = p + e. 

The analysis of the Bayesian AMP for Gauss -Bernoulli 
noise e (that is, with e = 0) has been considered in great 
details in lfTTlL [15]. In that case AMP reconstructs perfectly 
the solution in a region larger than the i\ -minimization and 
up to the so-called spinodal transition q;amp- In the notation 
of the present problem, where this leads exact decoding for 
considerably lower encoding rates: The resulting 7amp = 
1/(1 — a amp) is shown in red in Fig. [I] (with data adapted 
from [11]) where the advantage over the l\ -minimization 
decoding is clear. For a fraction of p = 0.1 gross elements, 
for instance, the improvement goes from a necessary coding 
rate 7 « 1.490 for t x to 7 « 1.262 for AMP. It should be 
noted, however, that the H\ performance is independent of the 
distribution of the gross error, whereas the Bayes AMP uses 
the distribution of the element of the gross error. The properties 
of the channel are, however, often well known, in which case 
the improvement depicted if Fig. [T] is indeed achievable. 

To assess how robust are these results towards approxi- 
mately sparse noisy channels (nonzero value of e in eq. {T])) 
we use the analysis that was performed in [12]. It was shown 
that for about e < 10 ~ 5 and a > a amp the AMP algorithm 
leads to reconstruction with MSE comparable to e. This shows 
that the AMP approach is actually very robust to such noise. 

IV. Spatially coupled measurements 

Despite the advantage of the AMP-based decoding over the 
H\ -minimization, it is still not asymptotically optimal since 
7amp > 7opt> and one ideally aims to perform error correction 
with smallest possible encoding rates. In order to do so, we 
shall mimic the strategy used in LDPC error correcting codes 
over binary signals (see [18] for a review) in compressed sens- 
ing based-codes over real- valued signals. It has been shown 
recently lfTTlL |[T3l (first heuristically and numerically, and later 
rigorously) that for compressed sensing of sparse signals with 
known empirical distribution of components the theoretically 
optimal reconstruction can be achieved in the large system size 
limit with the combined use of AMP algorithm HOlL lfT4l and 
seeding (spatially coupled) measurement matrices lfTTlL (HI. 

In the present framework of error correction of real- valued 
signals, the spatial coupling can be implemented by first 
constructing the matrix F as the seeding matrix of ifTTTl . 



Fig. 2. Sketch of the seeding matrix F we used to approach optimal 
reconstruction of the error vector with the AMP algorithm (see (l2))- We 
use a number of variable-blocks L c = L, and L r = L + 1 measurement 
blocks. The matrix components are iid with zero mean and variance 1/M for 
the blocks on the diagonal and for a number W of lower diagonals, the upper 
diagonal blocks have components with variance J/M. 



then determining the encoding matrix A as the null space 
of the matrix F. We refer to HH, El, EEL ED for a 
general discussion of spatial coupling in compressed sensing. 



The seeding measurement matrices F^ G 



(M-N)xN 



that 



we use in the rest of this paper are constructed as in 021, 
see Fig. [2] for an example. The M components of the error 
vector are sliced into L c equally-sized groups and the M — N 
measurements into L r groups, the first of which having a 
larger number of measurements m see d than the others rabuik- 
Define a see d = L c m seed /M and a hu}k = L c m hu i k /M. Define 
a = (M — N)/M as the total measurement rate, we have 
a = [a S eed + {L r — l)<^buik]/A- The matrix F is then 
composed of L r x L c blocks where the elements F^i are 
generated independently, such that if p is in group q and i 
in group p then F^i is a random number with zero mean 
and variance J qyP /M. In this work we use seeding matrices 
illustrated in Fig. [2] parameterized by L, W, and J. 

A rigorous analysis of the evolution of AMP reconstruction 
was performed for such matrices, assuming the knowledge of 
the L r x L c coupling matrix J q ^ p [17]. The state evolution for 
the block matrices can be done by defining El to be the mean- 
squared error in block p at time t. Then E^ 1 depends on rh p 
from the same block according to eq. (|16|). The quantity rh p 
depends on the MSE E l q from all the blocks q = 1, . . . , L c as 



-^seed^lp 
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r=2 l^q=l J rq^ q 



^L c 



(17) 



This kind of state evolution belongs to the class for which 
threshold saturation (asymptotic achievement of performance 
matching the optimal Bayes estimation of the error) was 
proven in [22] (when L — >• 00, W —> 00 and L/W ^> 1). 
It is also possible to compute the possible performance of 
B ayes-optimal reconstruction of the error analytically in the 
TV — >> 00 limit as done in |[T2lL and the result are shown again 
in Fig. [T] in green (lower-most) curve for e = and in red (3rd 
from top) curve for e = 10 -6 . The conclusion is that with the 
seeding matrices F, one can perform error correction using 
AMP down to these low coding rates. 



AMP, y=2, p=0.1 
convex, y=2, p=0.1 
AMP, y=1.5, p=0.1 

AMP, y=2, p=0.2 




Fig. 3. Robustness to noise of the error correction for signal of size TV = 
256. We compare the performances of the AMP-based and l\ decoders, using 
e = 10 _6 and homogeneous matrices F. The figure shows the probability 
density function (estimated with 500 instances) of the robustness ratio eq. fT8) , 
called Pideal in 1 8 1 at different values of the encoding rate 7 and gross noise 
sparsity p. For coding rate 7 = 2 and gross noise sparsity p = 0.1, both 
methods (AMP in red and i\ in black) are giving values close to one. However, 
AMP is better, on average it gives 1.12 versus 1.36 for l\. Furthermore, 
AMP still performs very well when the fraction of gross errors is double 
(blue curve, with p = 0.2) or when the coding rate 7 is lower (green curve, 
with 7 = 1.5). In both these cases, the l\ -based reconstruction gives poor 
results, an average value pideai — 37.1 for p = 0.2, 7 = 2, versus 1.30 for 
AMP; and p ide ai = 18.3 for p = 0.1, 7 = 1.5, versus 1.20 for AMP. 



V. Numerical tests 

The asymptotic guarantees given in the last sections are 
encouraging, but evaluating analytically finite M correction 
is intrinsically more difficult and hence we withdraw to 
numerical verifications of the achievable coding rates for sizes 
relevant for practical applications. Our Matlab code and demo 
are available on http://aspics.krzakala.org, 

For numerical verifications, we used Af -dimensional Gaus- 
sian signal x with zero mean and unit variance (the algorithm 
is not using this information) and a channel noise distributed 
according to eq. ([T]). We performed the Bayesian AMP al- 
gorithm to estimate the error e. For exactly sparse channel, 
e = 0, the exact reconstruction of e is possible and hence x 
can be recovered exactly. For approximately sparse channel, 
e > 0, we use the AMP estimate of the error e to compute the 
estimate of Ax and finally a pseudoinverse of A to estimate the 
signal x. We compare to the H\ decoding approach (including 
the reprojection step) as developed in [7], |8|. 

The data for A^ = 256 are shown in Fig. [3] where the perfor- 
mance of our Bayes AMP decoding algorithm is compared to 
the £i-based decoding of |8|. Following [8] we introduce an 
estimator of the robustness to noise called Pideai as the ratio 
of the MSE of the reconstructed signal x with the MSE of 
the "ideal" reconstruction Xid ea i» where the pseudoinverse of 
A is applied to y that was corrupted only by the small additive 
noise without gross errors 

■ X l|2 




Pideai = tt^ rr~ • (18) 

||Xideal -X|| 2 

Fig. [3] depicts the histogram of pideai over 500 random 



Y=M/N 

Fig. 4. Success rates of the decoding over 500 instances for different signal 
sizes N with noise parameters p = 0.1,e = 10 -6 , both with seeding matrices 
F (SC) and homogeneous random ones, as a function of the coding rate 
7. The vertical lines represent the limiting asymptotic coding rate for AMP 
with homogeneous and seeding matrices respectively. The maximum number 
of iterations in these simulations is set to 1000. An instance is considered 
as successful if the final mean-squared-error of the reconstructed signal is 
less than 10 -5 . The inner plot shows the probability of failure that decays 
exponentially with the signal size for three different values of encoding rate 7. 



instances of the problem. We find that in all the cases we 
have tried with AMP (which were all in the favorable region 
of the asymptotic phase diagram), the robustness estimator 
Pideai is very close to unity, even at these relatively small 
sizes. Moreover the robustness estimator of AMP was always 
on average closer to unity than the one based on i\ estimation 
and the distribution more peaked, thus demonstrating the 
advantage of the Bayesian AMP reconstruction in terms of 
performance, and noise robustness. Another important point 
is that the probability of a unsuccessful reconstruction is 
decaying exponentially fast when the system size increases. 
It is also decaying faster as 7 increases (see Fig. [4]). 

We also applied spatially coupled approach by choosing the 
matrix F as described in Fig. [2] with L = 10, W = 3, J = 
0.2, a see d = 0.22, and varying abuik- While such parameters 
are far from the limit L — >> 00, W —> 00, W/L —> in 
which the optimal performance is guaranteed, we still obtain 
a considerable improvement in the achievable encoding rate, 
as shown again in Fig. [4] In Fig. [5] we give a more visual 
illustration of the performance of the spatially coupled AMP 
decoder for N = 4096. For gross error sparsity p = 0.1 and 
small error variance e = 10 ~ 6 we were able to perform reliable 
error correction at coding rate 7 = 1.256. This has to be 
compared with the original approach of 1 8 ] which only allows, 
even for infinite system sizes and in absence of small noise, 
an asymptotic 7 « 1.5. 

VI. Conclusion and perspectives 

In this paper we considered an error-correcting scheme for 
real-valued signals over channels that disrupt the transmitted 
signal by a large error on a small fraction of elements. 
We combined a Bayesian AMP reconstruction and spatially- 
coupled decoding matrices. We show that this approach is 




No error-correction 



Fig. 5. Illustration of the error correcting scheme with the spatially coupled AMP approach and the t\ -based method of [81, applied to the benchmark Lena 
picture. The original 256 x 256 image is decomposed in patches of size TV = 64 2 . The noisy channel is given by eq. {T| using p = 0.1, e = 10 -6 and 
the coding rate is 7 = 1.256 (to be compared with 7opt(e = 10 -6 ) = 1.184) is used together with spatially-coupled (seeded) matrix F with parameters 
L = 10, W = 3, J = 0.2, a seed = 0.22, a bu ik = 0.1830. 



robust to non-sparse small noise, we computed the phase dia- 
gram in the limit of large signal sizes and showed numerically 
that the probability of failure decreases exponentially in the 
signal size. 

There is a number of possible improvements on the present 
approach. First, the AMP approach is parallelizable, thus 
allowing substantial gain in execution time. One could also 
use structured F matrices, for instance Fourrier, Gabor or 
Hadamard, as in (23), to further decrease the execution time 
of the AMP algorithm. Finally, another natural extension of 
the present work is when the signal x is itself compressible, 
so that a joint source and channel coding approach can be 
applied. We have done preliminary investigations on sparse 
signals in this direction. 
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